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Abstract 


The purpose of this study was to evaluate the program initiated by 
EdReports.org for analysing instructional materials, improving local decision 
making for selecting materials, and providing professional learning to 
support implementation of high-quality materials in schools. The Context 
Input Process Product model was used to evaluate decision-making by 
policy groups during the stage of context evaluation, and operations 
managers in EdReports.org during the stages of input, process and product 
evaluation. The results showed that EdReports.org reviews materials in 
English language arts, mathematics and science, and supports efforts to 
improve local selection and implementation of high-quality materials in 
schools. Activities include publishing research findings, increasing outreach 
within the education community, collaborating with education organisations 
and professional associations to strengthen professional learning for 
implementing high-quality materials, and partnering with state education 
agencies on projects to assist districts ensure that teachers use high-quality 
materials. The findings of the evaluation showed that EdReports.org has 
planned, structured and implemented a program during the first five years of 
operation that meets the needs of educators for guidance and advice about 
the selection and implementation of high-quality, standards-aligned 
materials. 


Introduction 


In 2009 and 2010, the NGA Center for Best Practices and the Council of Chief 
State School Officers (CCSSO) oversaw the development of the Common Core 
State Standards (CCSS) by content experts in English Language Arts and 
Mathematics. Released in June 2010, the CCSS were adopted by 46 states 
and the District of Columbia between 2010 and 2012. Opposition from 
conservative, anti-Common Core groups, which peaked in 2014, led 11 states 
to revise the CCSS to a major extent. From 2011 to 2013, Achieve oversaw 
content experts use a framework based on the three dimensions of scientific 
and engineering practices, crosscutting concepts, and disciplinary core ideas, 
published by the National Research Council (2012), to develop the Next 
Generation Science Standards (NGSS). Released in April 2013, the NGSS 
have been adopted by 20 states and the District of Columbia. 


The decade following adoption of the CCSS was characterised by three main 
trends in relation to instructional materials. Implementation of the CCSS led to 
a wave of research studies investigating the role of materials within the 
implementation process. Subsequently, policymakers launched several 
initiatives to evaluate the quality of lessons and units, print-based and digital 
materials and their alignment to the CCSS. In a spate of policy papers that 
followed these developments, education experts used the findings of the 
research studies to advance the policy position that implementation of materials 
needs professional learning support. 


EdReports.org, one of the initiatives launched by policy groups in 2014, forms a 
case study reported in this paper. The purpose of this study is to evaluate the 
program initiated by EdReports.org for analysing instructional materials, 
improving local decision making for selecting materials, and providing 
professional learning to support implementation of high-quality materials in 
schools. The evaluation investigates the program in terms of its genesis, review 
process, expansion during the first five years of operation, and the formation of 
partnerships with other education organisations and state education agencies. 
Based on these assumptions, the study focuses on four research questions. 
What conditions, needs and problems led to the foundation of EdReports.org? 
What action plan did EdReports.org initiate to review materials, report reviews, 
and provide staff, facilities and a budget to operate and maintain the program? 
What internal and external factors led EdReports.org to revise its program? 
What internal and external factors led EdReports.org to expand its program, 
and form partnerships with other education organisations and state education 
agencies? 


Method 


The research plan uses the Context Input Process Product (CIPP) model, first 
conceptualised as part of an evaluation project conducted by the Evaluation 
Center in Ohio State University for Columbus School District in 1966. Early in 
1968, the honorary professional fraternity, Phi Delta Kappa, formed a seven- 
member Study Committee on Evaluation, which formulated and published the 
CIPP model (Stufflebeam et al., 1971). Subsequently, the CIPP model was 


developed further, applied to evaluate educational programs in a wide range of 
settings and countries, and adapted for use in fields outside education. 


The CIPP model is used to assess whether a program addresses four stages of 
evaluation. Context evaluation, conducted by policy groups, assesses the 
conditions, needs and problems to specify planning decisions for the program. 
Input evaluation, conducted by operations managers, determines capabilities, a 
strategy and a design for implementing the strategy to specify structuring 
decisions for the program. Process evaluation, conducted by operations 
managers, identifies or predicts defects in the design to specify implementing 
decisions for the program. Product evaluation, conducted by operations 
managers, measures and interprets attainments to specify recycling decisions 
for the program. 


The methodology for context evaluation employs two modes. In the contingency 
mode, exploratory probing is conducted to promote improvement by identifying 
relevant research literature, assessing community values, conducting study 
visits, contracting external consultants, convening brainstorming retreats, and 
predicting future needs. In the congruence mode, the program’s mission and a 
statement of goals are defined in the form of operating guidelines. The 
methodology for input evaluation varies depending on whether large or small 
change is involved or whether a high or low information base is available to 
support the change. Choice between alternative types of change is determined 
according to the decision setting: homeostatic, where small change is supported 
by much information; incremental, where small change is supported by little 
information; neomobilistic, where large change is supported by little information; 
or metamorphic, where complete change is supported by much information. 
Since the metamorphic setting is utopian and essentially theoretical, each of the 
other decision settings uses a different model. The synoptic ideal model applies 
to the homeostatic setting. The model of disjointed incrementalism applies to 
the incremental setting. The planned change model applies to the neomobilistic 
setting. The methodology for process evaluation uses various formal and 
informal procedures to collect information to provide feedback to operations 
managers about defects in the design. The methodology for product evaluation 
uses experimental and quasi-experimental designs to provide feedback to 
operations managers on whether objectives are being, or have been, attained. 


The CIPP model is used in the present study to evaluate decision-making by 
policy groups during the stage of context evaluation, and operations managers 
in EdReports.org during the stages of input, process and product evaluation. 
The research design is qualitative in nature, since it comprises content analyses 
of verbal communications and written documents with the intention of 
uncovering information to answer the research questions. 


The procedures for collecting information for the evaluation involved reviewing 
resources published on the website of the Annenberg Foundation Trust at 
Sunnylands, working with EdReports.org operations managers, and reviewing 
resources published on the EdReports.org website. Information about the 
foundation of EdReports.org, identified on the website of the Annenberg 
Foundation Trust at Sunnylands, included a news article, Revolutionising K-12 
Mathematics, and the videos, Teaching Mathematics in K-12 and Connected 
Learning in the Digital Age: Digital Learning Content. The EdReports.org chief 


strategy officer and data strategist were consulted, and email alerts, news 
articles, reports on case studies, an evaluative report, annual reports, opinion 
papers, frequently asked questions and instructional material evaluations were 
reviewed, and videos presented by reviewers were viewed. 


This procedure was extended to work with program managers in five state 
education agencies, which had established programs for analysing instructional 
materials in partnership with EdReports.org. In addition, various resources 
about these programs were reviewed. Furthermore, staff of CCSSO was 
consulted to provide information about the Instructional Materials Professional 
Development Network, which provided the basis for these partnerships. 


The Massachusetts Department of Elementary and Secondary Education’s 
manager of instructional policy was consulted, and the Curriculum Ratings by 
Teachers (CURATE) portal and the CURATE Curriculum Fellowship published 
on Teach Plus Massachusetts’s website were reviewed. The Mississippi 
Department of Education’s bureau director for secondary education was 
consulted, rubrics and evidence guides were reviewed, and a _ webinar 
presented by Marla Davis was viewed on the High Quality Instructional 
Materials webpage in the Mississippi Department of Education’s website. The 
Nebraska Department of Education’s chief academic officer was consulted, 
webpages on the Nebraska Instructional Materials Collaborative website were 
reviewed, and webinars presented by Cory Epler and Marissa Payzant as well 
as publishers’ representatives for Amplify Science, ReadyGEN, Wit & Wisdom, 
ARC Core, EL Education and Bookworms were viewed. Reports on state-wide 
surveys, professional learning plans and a report setting out a state plan were 
reviewed, and videos, presented by David Steiner, Eric Hirsch and Lauren 
Weisskirk, David Liben, Kate Gerson and Emily Freitag, were viewed on the 
Instruction and Assessment webpage in the Rhode Island Department of 
Education’s website. The Wisconsin Department of Public Instruction’s literacy 
consultant was consulted, a report on a survey of teachers was reviewed, and 
videos presented by Sheila Briggs, Kate Gerson, Eric Hirsch and the Wisconsin 
Educator Panel were viewed on the Instructional Materials and Professional 
Learning webpage of the Wisconsin Department of Public Instruction’s website. 


The procedure for analysing information, collected during the study, used 
content analysis method to summarise the subject matter focusing on 
interpretation of relevant resources. In-depth analysis of the EdReports.org 
program and the programs of state education agencies focused on analysing 
the subject matter of relevant resources and summarising the findings. Finally, 
judgments were reached concerning the nature of the decision-making process 
during each stage of the evaluation. 


EdReports.org 

Origins and structure 

The difficulty that educators experience in identifying high-quality materials 
prompted Maria Klawe, president of Harvey Mudd College at Claremont, 


California, to convene a Sunnylands Mathematics Strategy Group at the 
Annenberg Foundation Trust at Sunnylands in Rancho Mirage, California, to 


examine opportunities that digital content brings to helping teachers and 
students become more effective in mathematics education. In 2012 and 2013, 
the Sunnylands Mathematics Strategy Group, consisting of mathematics 
educators, digital content specialists, lead writers of the CCSS and education 
leaders, held three design meetings and a culminating retreat to discuss the 
need for independent reviews of materials focusing on their alignment to the 
CCSS and concluded that an entity should be established to pursue this work. 
Grants were received from several philanthropic foundations to establish a non- 
profit organisation. Education First, a Seattle-based consulting group, 
contracted by the funders to plan the new organisation, recruited a Board of 
Directors, developed a business plan, hired an executive director, housed the 
organisation, provided technical support and launched a website at 
www.edreports.org. 


Since its foundation in July 2014, EdReports.org has established a middle-sized 
organisation with grants provided by the Bill and Melinda Gates Foundation, 
Broadcom Incorporated, Carnegie Corporation of New York, Charles and Helen 
Schwab Foundation, Charles and Lynn Schusterman Family Foundation, Leona 
M. and Harry B. Helmsley Charitable Trust, Oak Foundation, Overdeck Family 
Foundation, Samueli Foundation, Stuart Foundation, and William and Flora 
Hewlett Foundation. EdReports.org is governed by a twelve-member Board of 
Directors, whose members have professional backgrounds in education, 
finance, law, non-profit management, and marketing and communications. 


Since the appointment of an executive director in 2014, the number of staff has 
increased to 32 employees. EdReports.org is operated by an executive director, 
a chief operating officer, a chief strategy officer, a chief academic officer, an 
academic coordinator, a knowledge manager specialist, six content specialists, 
an early literacy consultant, three directors of review, four instructional materials 
managers, a program manager, an operations coordinator, a director of field 
services, a director of communications, a communications manager, two 
outreach specialists, a human resource manager, a finance manager, a 
managing editor, a data strategist and a full stack developer. EdReports.org is a 
virtual organisation with staff working in home offices using communications 
technology to support collaboration. Staff members, however, meet at annual 
retreats as well as board meetings, workshops and reviewer trainings. 


EdReports.org maintains a network of more than 500 content reviewers 
representing education systems in urban, suburban and rural communities 
across 47 states and the District of Columbia. After completing an interview and 
a work sample, prospective reviewers are selected according to depth of 
content knowledge, experience at evaluation, and ability to participate in face- 
to-face meetings and virtual conferences, but they must not be affiliated with the 
publishing industry. Each summer, newly recruited reviewers participate in 
annual training sessions, which were held in Chicago from 2014 to 2017 and in 
Minneapolis in 2018 and 2019. At the training sessions, reviewers extend their 
knowledge about college- and career-readiness standards, the instructional 
shifts, and the EdReports.org review process. Reviewers are paid a stipend at 
the end of each review. In October 2019, EdReports.org appointed eight current 
and former reviewers as inaugural Klawe fellows to develop and implement 
impact projects in their communities in 2020 that support the mission of 
EdReports.org. 


Review and selection process 


EdReports.org staff assessed 11 commonly used rubrics, and observed review 
processes and training conducted for Achieve and the state of Tennessee to 
develop a process for reviewing digital and print-based materials. The review 
process requires materials to meet criteria and indicators set for three 
successive gateways: alignment to the CCSS or NGSS; rigour of the subject 
matter; and instructional supports and usability. At each gateway, reviewers 
provide a rating according to numerical rating options and cite concrete 
evidence in an evidence collection workbook. An overall rating for each criterion 
is determined by adding the total points earned from rating a material against 
the indicators. A material that meets expectations moves to the next gateway. 


To identify materials for review, EdReports.org staff researches the 
marketplace, accepts recommendations from educators and receives requests 
from publishers. The review process involves teams of four or five reviewers 
analysing each material independently and providing evidence through the 
online system before meeting to reach consensus on the evidence and the 
score. Review teams are assisted by volunteer advisory panels with expertise in 
each content area. 


Publishers are also involved in the review process. Each publisher is invited to 
provide an hour-long orientation on the publisher’s material to the appropriate 
review team. Publishers also have opportunities to post a response to a review 
report in the form of a document providing background information and research 
findings about the material for publication on the EdReports.org website. 


In April 2020, EdReports.org released modifications to the rubrics used in the 
review process. Revisions to a small number of indicators, focusing on 
alignment to the CCSS and rigour of the subject matter, were made to the 
rubrics for English language arts and mathematics. Revisions to indicators, 
focusing on instructional supports and usability, were made to the rubrics for 
English language arts, mathematics and science to address details for student 
populations that require support for English language acquisition, learner 
variance, and teacher supports. Reviews using the revised rubrics commenced 
in mid-2020 with the first rounds of reports to be published early in 2021. 


Mathematics 


Early in 2014, EdReports.org staff conducted a listening tour to collect 
information for developing a methodology and a rubric to analyse mathematics 
materials for kindergarten to grade 8. An Anchor Educator Working Group 
analysed the feedback and developed the Quality Instructional Materials Tool: 
K-8 Mathematics and K-8 Mathematics Evidence Guides. Following a 
calibration exercise to ensure consistency across reviewers, 20 materials for 
kindergarten to grade 8 were analysed by 47 reviewers. EdReports.org posted 
the results in March 2015. 


Following criticism of the gateway procedure by several publishing companies 
and release of an open letter by the National Council of Teachers of 
Mathematics and the National Council of Supervisors of Mathematics outlining 
concerns about the review process, EdReports.org released four enhancements 
to the rubric, methodology and reporting protocol. Then, the review teams 
analysed the materials again as well as an additional 58 materials. Evaluations 
of these materials were released on a rolling basis in February, April and May of 
2016. 


Early in 2015, EdReports.org conducted a listening tour to collect information for 
developing a methodology and a rubric to analyse mathematics materials for 
high school. An Anchor Educator Working Group analysed the feedback and 
developed the Quality Instructional Materials Tool: High School Mathematics 
and High School Mathematics Evidence Guides. Late in 2015, 31 reviewers 
evaluated eight traditional course materials for algebra |, geometry and algebra 
Il as well as integrated course materials. EdReports.org released the results for 
four traditional course materials and one integrated course material in June 
2016. After analysing an additional nine materials, EdReports.org posted the 
results in October 2016. 


English Language Arts 


In mid-2015, EdReports.org conducted a listening tour to collect information for 
developing a methodology and a rubric to analyse English language arts 
materials for grades 3 to 8. An Anchor Educator Working Group analysed the 
feedback and developed the EdReports.org Quality Instructional Materials Tool: 
ELA 3-8 Review Tool and 3-8 ELA Quality Instructional Materials Tool Evidence 
Guides. Late in 2015, 45 reviewers analysed seven English language arts 
materials for grades 3 to 8. In August 2016, EdReports.org posted the results. 
In September 2016, EdReports.org held a webinar to report the findings of the 
review. 


In 2016, EdReports.org conducted a listening tour to collect information for 
developing a methodology and a rubric to analyse English language arts 
materials for kindergarten to grade 2. An Anchor Educator Working Group 
analysed the feedback and developed the EdReports.org Quality Instructional 
Materials Tool: ELA K-2 Review Tool and K-2 ELA Quality Instructional 
Materials Tool Evidence Guides. Early in 2017, review teams analysed six 
English language arts materials for kindergarten to grade 2. In April 2017, 
EdReports.org posted the results. 


In 2017, EdReports.org conducted a listening tour to collect information for 
developing a methodology and a rubric to analyse English language arts 
materials for high school. An Anchor Educator Working Group analysed the 
feedback and developed the EdReports.org Quality Instructional Materials Tool: 
ELA High School Review Tool and High School ELA Quality Instructional 
Materials Tool Evidence Guides. Late in 2015, six review teams analysed six 
English language arts materials for high school. EdReports.org posted the 
results in August 2017. 


A national conversation over the effectiveness of the balanced literacy approach 
for reading instruction led EdReports.org to conduct a listening tour to collect 
information for developing a methodology and a rubric to analyse supplemental 
materials for foundational skills in literacy. An advisory panel analysed the 
feedback and developed the EdReports.org Quality Instructional Materials Tool: 
English Language Arts Foundational Skills Review Tool and ELA Foundational 
Skills Evidence Guides. Late in 2019, review teams analysed five supplemental 
materials for foundational skills in literacy. EdReports.org posted the first reports 
in November 2019. 


Science 


In November 2017, EdReports.org conducted a listening tour involving science 
education experts to collect feedback that ensured that the vision of the 
framework, published by the National Research Council (2012), and the NGSS 
were reflected in a methodology and a rubric to analyse science materials for 
grades 6 to 8. In January 2018, an Anchor Educator Working Group analysed 
the feedback and developed the Science Quality Instructional Materials Rubric: 
Grades 6-8 and the Middle School Science Evidence Guide. Beginning in May 
2018, review teams reviewed six science materials for grades 6 to 8. 
EdReports.org posted the results in February 2019. 


Reports 


In May 2019, EdReports.org (2019a) published its first annual report. The report 
describes the origins of EdReports.org, inaugural reviews of mathematics 
materials and expansion to analysing English language arts and science 
materials, compares the scope of the organisation’s activities between March 
2015 and the end of 2018, outlines the organisation’s theory of action for 
identifying excellence in materials and increasing demands from states, districts 
and teachers for high-quality materials, lists the Board of Directors, and 
presents a financial report. 


In July 2019, EdReports.org published the findings of a study to identify what 
percentage of English language arts and mathematics materials, marketed as 
standards-aligned, meet EdReports.org criteria for alignment, what proportion of 
those materials used frequently meet expectations for alignment, and whether 
there is a relationship between the length of time that an EdReports.org 
evaluation has been available for a material and its proportion of market share. 
Data on materials reviewed by EdReports.org between March 2015 and April 
2018 were analysed to answer the first research question. Data from the 2017- 
2018 RAND American Educator Panels Survey were analysed to answer the 
second research question. A multivariate, multilevel analytic framework was 
used to analyse data from the 2017-2018 RAND American Educator Panels 
Survey and EdReports.org standards-alignment ratings of reviewed materials to 
answer the third research question. 


EdReports.org (2019b) reported the results of the study and suggested 


extensions for further research. Of the 472 materials reviewed by 
EdReports.org during this period, 49 percent of English language arts and 28 


10 


percent of mathematics materials met expectations for alignment with newer 
materials more likely to meet these expectations. However, only 15 percent of 
materials regularly used by English language arts teachers and 23 percent of 
materials regularly used by mathematics teachers met expectations for 
standards alignment. On the other hand, increasing age of a report's availability 
for a material that met expectations for standards alignment was associated 
with a one percent increase in teachers, who reported using aligned materials at 
least once a week. Furthermore, increasing age of a report’s availability for a 
material that did not meet expectations for standards alignment was associated 
with a one percent decrease in teachers, who reported using aligned materials 
at least once a week. As well as extending statistical analysis into other aspects 
identified in the results of the study, the project lead concluded research needs 
to be undertaken into materials available online and a database of information 
on the adoption and procurement of materials needs to be developed to 
facilitate further research into the effects of materials on student learning 
outcomes. 


In March 2020, EdReports.org (2020) published its second annual report. The 
report sets out impact data, reviewer data, a vision for 2020 and beyond, and 
additional information. Impact data report the number of instructional reviews 
published, documented districts using EdReports.org, largest districts 
documented using EdReports.org, school leaders who have heard of 
EdReports.org, district leaders who have heard of EdReports.org, collaborations 
with state departments of education, EdReports.org total page views, and 
publishers that have changed their materials in response to EdReports.org 
reviews. Reviewer data report the number of reviewers, reviewers’ average 
years in education, reviewers’ total years of experience in education, hours 
reviewers spent conducting reviews in 2019, and the state of the instructional 
materials market. The vision for 2020 and beyond outlines intentions to release 
initial reports in science for kindergarten to grade 5 and expanding partnerships 
with states and districts. Additional information presents financial statements, 
lists funders and specifies members of the Board of Directors. 


Productivity and outreach 


By May 2019, EdReports.org had reviewed more than 90 percent of textbooks, 
digital materials and open educational resources available for English Language 
Arts and Mathematics. More than 880 districts across 48 states, enrolling over 
11,000,000 students, use EdReports.org reports to select materials. Reviews of 
materials had led 30 publishing companies to improve their products. 


In October 2018, EdReports.org launched a revamped website, which is 
organised into five areas. Compare Materials presents the reports of reviews in 
English Language Arts for kindergarten to grade 2, grades 3 to 8, and high 
school, Mathematics for kindergarten to grade 8, and high school, and Science 
for grades 6 to 8 in a form that allows users to compare text quality, building 
knowledge, alignment rating and usability rating. Reports Center presents the 
reports of reviews as a database searchable by publisher, subject grade or 
report title, upcoming reviews, and rubrics and evidence guides. Resources 
present various resources relating to EdReports.org activities. Impact presents 
information about productivity and outcomes for reviewers, annual reports and 
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research reports. About Us presents biographies about EdReports.org staff, 
Board of Directors and reviewers, the review process, recruitment of staff and 
reviewers, frequently asked questions, and contact details. 


In 2015, Editorial Projects in Education, the publisher of the online newspaper, 
Education Week, launched EdWeek Market Brief to report research findings 
focusing on district purchasing of educational products and services, and the 
companies and products servicing them. In February 2019, EdWeek Market 
Brief published an article about the EdReports.org review process based on an 
interview with EdReports.org Executive Director, Eric Hirsch. In July 2019, 
EdWeek Market Brief hosted a webinar, in which Eric Hirsch outlined research 
findings about the positive effects of materials on student achievement, and 
EdReports.org processes for reviewing materials, training reviewers and 
engaging with publishers. Sonja Santelises, Baltimore City Public Schools chief 
executive officer and EdReports.org board member, outlined how this district 
uses EdReports.org reviews. 


Subsequently, EdReports.org (2019c) composed a case study describing an 
audit conducted by Baltimore City Public Schools to identify teachers’ use of 
materials. The findings of the survey led to the introduction of a new process for 
adopting a new English language arts material involving a committee screening 
available materials followed by stakeholder engagement to select one of two 
materials. Poor participation of teachers in feedback sessions led the district to 
host a Facebook Live session for the local community. Following adoption of the 
new material in June 2018, its implementation included professional 
development sessions for district staff, school leaders, teachers and literacy 
coaches. 


Partnerships with states 
California initiatives 


In mid-2015, EdReports.org began working with Orange County Department of 
Education, which provides support services to 27 districts in southern California. 
Mathematics coordinators, Jody Guarino and Vanessa Cerrahoglu offered to 
become partners in an effort to provide better guidance and support using 
EdReports.org ratings by developing a series of training sessions and supports 
delivered to three districts. In September 2015, superintendents, coaches and 
teachers from three districts and one charter school met in Westminster to use 
EdReports.org ratings to screen available materials and pilot selected materials 
in classrooms. 


One of these districts, Newport-Mesa Unified School District, consisting of 32 
schools located in Newport Beach, Costa Mesa and Corona del Mar, asked 
Orange County Department of Education for more intensive support to adopt a 
new mathematics material. Guarino, Cerrahoglu, Drake and Weisskirk (2018) 
reported that the adoption process, including professional learning, consisted of 
three phases: ground the work in shared understanding of the standards; apply 
the learning to two cycles of materials evaluation; and build consensus. A 
steering committee was trained to screen available materials, publishers were 
involved in the final selection of two materials, 30 lead pilot teachers were 
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trained, and 123 teachers piloted the materials. Since evidence collected during 
the pilot was inconclusive, a consensus process was used to reach agreement 
about adoption of one material. Embodied in the decision-making process, a 
teacher-centred approach assisted implementation of the adopted material. 
Afterwards, EdReports.org (2018) composed a case study outlining the 
adoption process. 


Based on the success of the collaborative work with Orange County 
Department of Education, EdReports.org decided to customise its work to meet 
the needs of California's large materials’ marketplace and the provision, 
introduced in 2013, allowing districts to use materials not adopted by the State 
Board of Education. Eric Hirsch met staff of Pivot Learning, a non-profit 
organisation based in the San Francisco Bay Area at a grantee conference held 
by the William and Flora Hewlett Foundation. This meeting led EdReports.org to 
form a partnership with Pivot Learning to provide independent reviews to help 
teachers identify high-quality materials for use in California schools. The two 
organisations raised funds for the venture in 2016, and developed the 
components for a new website, California Curriculum Collaborative, launched at 
www.calcurriculum.org in February 2017. Following the launch, the two 
organisations hosted a series of regional workshops to guide district teams to 
use the rubric and processes on the website. Funded by the Bill and Melinda 
Gates Foundation, the two organisations added guidance to the website for 
Science and English language learners. In 2018, workshops were conducted for 
districts with high proportions of students from low socio-economic and minority 
backgrounds. Early in 2019, workshops were held to discuss the 2018 state 
adoption of science materials and EdReports.org reviews of science materials. 
In May and June of 2019, workshops were held at Sacramento and Santa Ana 
to help staff of county offices of education, district administrators and staff of 
charter management organisations to identify opportunities and strategies for 
implementing mathematics materials. 


The California Curriculum Collaborative is organised into six areas. About 
presents information about the partnership between EdReports.org and Pivot 
Learning as well as the findings of research studies about the effects of 
materials on student learning. News present news articles about the California 
Curriculum Collaborative. Resources present links to professional resources. 
Reports set out the EdReports.org review process, annotations for kindergarten 
to grade 8 and high school materials for English Language Arts, Mathematics 
and Science containing EdReports.org reviews and indicators of state adoption. 
Implementation presents an implementation guide, published by the California 
Curriculum Collaborative (n.d.), which sets out a six-step implementation 
process consisting of pre-work, discover, prioritise, ideate, plan and prototype, 
and support for implementing mathematics materials. Adoption presents 
guidance consisting of pre-work, prioritise, select and rollout, and key questions 
and guidance for adopting materials for English Language Arts, Mathematics 
and Science at the local level. 


Florida initiative 


In 2014, TNTP, an organisation that recruits and trains teachers for schools in 
disadvantaged urban communities, partnered with three districts in Florida to 
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assess their implementation of the Florida Standards. Collection and analysis of 
data identified challenges to implementation arising from inconsistent 
professional development, and misaligned curricula and materials. In 2016, 
TNTP formed the Pilot Florida Implementation Network consisting of six 
districts: Brevard; Broward; Duval; Highlands; Pasco; and Pinellas. The six 
districts collaborated by establishing network goals, sharing work, and 
designing individual district work focusing on a few key areas. Focus on 
materials led the districts to request EdReports.org to support district reviews of 
mathematics materials under consideration for state adoption in 2018-2019 
prior to revision of the adoption cycle following an executive order, issued by 
Governor Ron DeSantis in January 2019, requiring a standards review that led 
to the adoption of Florida’s Benchmarks for Excellent Student Thinking 
Standards in February 2020. EdReports.org released reviews of Florida editions 
for 15 mathematics materials between January and June of 2019. 


Instructional Materials Professional Development Network 


In 2016, CCSSO’s Chief Academic Officer Network met in Louisiana and 
examined the Louisiana Department of Education’s initiative to assist districts 
ensure that teachers use high-quality materials. Network members visited 
schools, interviewed administrators and teachers, and learnt about Louisiana’s 
policy to provide districts and schools with detailed information and tiered 
rankings of materials and professional development providers. The Chief 
Academic Officer Network decided to adapt this model for use in their own 
states and asked CCSSO to support them. In February 2017, CCSSO created 
the Instructional Materials Professional Development Network to support 
Delaware, Massachusetts, Mississippi, Nebraska, New Mexico, Rhode Island, 
Tennessee and Wisconsin initiate state projects. CCSSO periodically convenes 
meetings for teams from the eight states to provide opportunities for cross-state 
collaboration, discussions with experts, feedback on state plans and challenges, 
and time to work on state plans. 


EdReports.org collaborated with state education agencies in Massachusetts, 
Mississippi, Nebraska, Rhode Island and Wisconsin to support their projects by 
providing technical assistance. 


Massachusetts 


The Massachusetts Department of Elementary and Secondary Education 
collaborated with Teach Plus Massachusetts and the Rennie Center for 
Education Research and Policy to convene panels of Massachusetts educators 
to review materials for the Curriculum Ratings by Teachers (CURATE) project. 
Contracted to support the CURATE project, EdReports.org presented webinars 
on its review process and developed a strategy to involve publishers in the 
review process. Materials in English Language Arts for kindergarten to grade 5, 
Mathematics for grades 6 to 8, and Science and Technology-Engineering for 
grades 6 to 8 were reviewed in the 2018-2019 school year. Materials in English 
Language Arts for kindergarten to grade 8 and Mathematics for grades 9 to 12 
were reviewed between September and December of 2019. Materials in English 
Language Arts for grades 3 to 8, Mathematics for kindergarten to grade 5 and 
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grades 9 to 12, and Science and Technology-Engineering for grades 6 to 8 
were reviewed between February and May of 2020. The CURATE project will 
be expanded gradually to include other grade levels. 


Following selection as a CURATE curriculum fellow, each educator is trained to 
use the rubrics and review process before reviewing document evidence 
independently. Working in panels, the educators evaluate, calibrate and rate the 
materials against specific criteria, and meet at panel sessions on four to six 
occasions. The review process involves publishers submitting their products 
followed by an initial screen of each product to determine whether it is a 
comprehensive core material and has been found by an independent, teacher- 
based review process, such as EdReports.org, to be fully or partially aligned to 
the college- and career-readiness standards set out in the Massachusetts 
Curriculum Frameworks. Each panel considers the findings from _ the 
independent review, information supplied by the publisher, and data collected 
by surveys from or interviews with Massachusetts educators in reaching a 
consensus on the ratings. Staffs from the Department of Elementary and 
Secondary Education and the Rennie Center for Education Research and 
Policy draft and refine a report, which is checked by the panel before being 
shared with the publisher. Each publisher is given an opportunity to respond to 
a report, which is published as an addendum. 


In September 2019, the Department of Elementary and Secondary Education 
launched a CURATE portal to house a policy brief, the review process, and 
resources consisting of reference guides, rubrics, maps showing the use of 
particular materials by Massachusetts districts and links to external websites. 
The CURATE portal presents overviews of ratings for English language arts and 
literacy materials, mathematics materials, and science and_ technology- 
engineering materials. CURATE reports present an overview, standards 
alignment, classroom application, and a publisher's report for each material. 


In 2019, the Department of Elementary and Secondary Education disseminated 
the CURATE program across Massachusetts. In February, grants were offered 
for districts to support adoption of high-quality materials by forming a curriculum 
council, providing a plan for decision-making and a preliminary plan for 
professional development or implementing high-quality materials by holding 
professional development sessions. Beginning in September, a series of 
workshops was conducted for district teams to partner with EdReports.org and 
Department of Elementary and Secondary Education staffs to identify curricular 
needs for English language arts, mathematics or science, evaluate materials, 
engage in the selection process, and develop implementation plans. In 
December, the Rennie Center for Education Research and Policy held an 
event, High-Quality Curriculum: a Foundation for Student Success. This event 
featured a keynote address by Commissioner Jeffrey Riley, an opportunity to 
learn how districts are identifying and using high-quality materials, and a panel 
discussion by Massachusetts educators. 


Mississippi 


Late in 2017, the Mississippi Department of Education formed a High-Quality 
Instructional Materials working group, which designed an implementation plan 
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consisting of a state-wide awareness campaign, revision of the textbook 
adoption process, and professional learning opportunities. The Department of 
Education worked with institutions of higher education teacher preparation 
programs to align expectations, launched a web page, and initiated a three-year 
pilot study with a small number of districts across the state. 


In March 2018, the Department of Education convened a summit at Jackson, 
Mississippi, for district superintendents, curriculum coordinators, and higher 
education faculty focusing on the High-Quality Instructional Materials and 
Professional Learning initiative. Keynote speakers, David Steiner from Johns 
Hopkins University and Rebecca Kockler from the Louisiana Department of 
Education, discussed research findings and provided expert advice on why the 
initiative should increase teacher capacity, improve student outcomes, and 
expand equitable access in Mississippi schools. The Department of Education 
worked with EdReports.org to develop mathematics rubrics for kindergarten to 
grade 8 and high school, train mathematics reviewers, and support the pilot 
districts. 


The Department of Education sought 18 districts to serve as pilot sites to 
strengthen the materials review and selection process. Selected teachers in the 
pilot districts receive intensive training on the Mississippi Career- and College- 
Readiness Standards, review mathematics materials, and participate in 
developing a comprehensive list of mathematics materials. Teachers are also 
participating in periodic meetings and on-site observations with Department of 
Education staff regarding district-wide implementation of mathematics materials. 
Future work includes collaboration with institutions of higher education to 
provide opportunities for pre-service teachers to complete internships within the 
district and extend opportunities for professional learning and observation to 
teachers within the district, neighbouring districts, and institutions of higher 
education. As an initial step in the pilot study, the Department of Education 
worked with Student Achievement Partners, an organisation established by lead 
writers of the CCSS, to conduct a three-day institute on the mathematics shifts 
and standards for curriculum coordinators, administrators, and lead teachers 
from eight districts. 


Nebraska 


In 2017, the Nebraska Department of Education, in partnership with the state’s 
17 educational service units, began a process to identify the most commonly 
used materials in Nebraska schools. With support from EdReports.org, the 
Department of Education designed a state-specific website that includes 
resources to ensure that materials meet the expectations of Nebraska’s 
College- and Career-Ready Standards. Following the launch of the website in 
August 2018, EdReports.org partnered with the Department of Education and 
the educational service units to train the staffs of the educational service units to 
support districts use the website. In September 2018, the Nebraska 
Instructional Materials Collaborative hosted a webinar to introduce Nebraska 
educators to issues relating to high-quality aligned materials, EdReports.org 
review process, and features of the website. In March 2019, the Nebraska 
Instructional Materials Collaborative hosted webinars involving publishers’ 
representatives and teachers about implementing materials for English 
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language arts and science followed by a webinar focusing on the selection of 
high-quality materials. 


The Nebraska Instructional Materials Collaborative website is organised into 
five areas. ELA presents EdReports.org reviews searchable by grade, eight 
steps for selecting materials, and links to external resources. Math presents 
EdReports.org reviews searchable by grade, eight steps for selecting materials, 
and links to external resources. Science presents EdReports.org reviews 
searchable by grade, eight steps for selecting materials, and links to external 
resources. Nebraska Resources define key terms, provide links to events, 
present archived webinars, set out eight steps for selecting materials and links 
to selection resources, and present a toolkit consisting of communications 
resources, reports and frequently asked questions. About presents a rationale 
statement for the Nebraska Instructional Materials Collaborative and contact 
details. 


Rhode Island 


In 2016, the Rhode Island General Assembly passed An Act Relating to 
Education - The Rhode Island Board of Education Act, requiring a 
comprehensive study of a unified approach to state-wide education. In the 
report on the study, the Rhode Island Department of Education (2017) found 
from a state-wide survey of educators that they did not have a shared definition 
of curriculum, spent excessive time searching for materials, and believed 
materials they used could be improved. The report recommended developing a 
consistent curriculum for English language arts and mathematics, and enabling 
districts to identify high-quality materials and plan for implementation. The 
report’s recommendations led the Department of Education to establish a state- 
wide curriculum and professional learning initiative consisting of four strategies. 


In 2017, the Division of Teaching and Learning began a series of district 
network meetings on curriculum. In October 2017, David Steiner discussed 
research findings indicating that high-quality materials affect student learning. In 
January 2018, Eric Hirsch, Lauren Weisskirk and Audra McPhillips presented 
an overview on the work of EdReports.org. In May 2018, David Liben of Student 
Achievement Partners discussed the role of high-quality English language arts 
materials for developing independent and proficient readers. In June 2018, Kate 
Gerson of UnboundEd, a consulting service providing standards-aligned 
resources and professional development to teachers, discussed the importance 
of standards-based education in increasing equity for disadvantaged ethnic 
groups. In October 2018, Emily Freitag of Instruction Partners, a consulting 
service that works with districts to align curriculum to standards, discussed 
common pitfalls for implementing materials and presented a framework for 
effective implementation. 


After working with Rhode Island districts in 2016, the Department of Education 
contracted EdReports.org to assist 16 districts access and use information 
about the quality of mathematics and science materials during the adoption 
process. 
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The Department of Education conducted two state-wide surveys to identify what 
materials districts were using for reading and mathematics for kindergarten to 
grade 8. In 2017-2018, Eureka Mathematics and EngageNY were the most 
commonly used materials in mathematics although ten districts used locally 
developed materials, while the majority of districts used locally developed 
materials for reading. Of 52 districts that responded to the survey in 2018-2019, 
23 districts were using mathematics materials and seven districts were using 
reading materials rated as high-quality by EdReports.org. Another seven 
districts are expected to adopt high-quality mathematics materials and another 
eight districts are expected to adopt high-quality reading materials in 2019- 
2020. 


In 2019, the Department of Education released two professional learning plan 
templates and samples to support districts to meet certification regulation 
requirements beginning in 2019-2020. The existing plan, designed for districts 
that already have a professional learning plan, and the comprehensive plan, 
designed for districts that do not have a professional learning plan, provide the 
basis for districts to specify four types of professional learning: enhance 
educator effectiveness; build curricular implementation knowledge; develop 
pedagogical knowledge; and deepen content knowledge. The professional 
learning plans can be used for implementing newly adopted materials. 


Wisconsin 


In mid-2018, the Wisconsin Department of Public Instruction commissioned the 
Center for Public Research at Columbia University to conduct a series of focus 
groups with teachers and instructional coaches to learn their perceptions about 
defining, selecting and accessing high-quality materials and associated 
professional learning, issues that resonate or concern them in these areas, and 
steps that should be taken to support them. The methodology involved 
interviewing eight key state officials, conducting focus group sessions with 79 
educators at five cooperative educational service agencies, and using codes for 
qualitative analysis of focus group data. 


Cahn, Blancato and Yoon (2018) reported the findings in relation to seven 
categories: content of materials; selection of materials; content of professional 
learning; selection of professional learning; equity; messaging; and 
communications. For the first category, subjects wanted materials that align to 
the Wisconsin Academic Standards, were sceptical of the quality of commercial 
materials and had greater faith in self-developed materials, sought to balance 
availability of materials with time to identify them, wanted materials that are 
useful for differentiation, and varied in their opinions about materials that are 
directive. For the second category, variability prevailed in how materials were 
selected and the extent to which teachers were involved in the decision-making 
process, the process was based on screening available materials to produce a 
short list, and decision-making was affected by factors that are not related to the 
quality of materials. For the third category, subjects preferred direct professional 
learning delivered by providers with real and recent teaching experience, and 
ongoing professional learning that allowed for implementation of the learning. 
For the fourth category, variability prevailed in how districts select professional 
learning providers, but teachers had greater input into selection decisions, and 
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they wanted professional learning that was differentiated based on their needs. 
For the fifth category, subjects with greater familiarity with issues of equity were 
more vocal about its importance, and awareness of equity issues was highest 
among subjects, who received support from the Department of Public 
Instruction. For the sixth category, subjects found certain concepts motivating, 
and several key concepts resonated as characteristics of high-quality materials. 
For the seventh category, subjects cited being most influenced by other 
teachers they knew, and reported having contacts with the Department of Public 
Instruction mainly in relation to licensure, accountability and data analysis. 


In November 2018, the Department of Public Instruction hosted a one-day 
professional learning event, Quality Instructional Materials: Unlocking Teacher 
Creativity and Increasing Equity, for over 400 educators, consisting of school- 
based teams convened at the state’s 12 cooperative educational service 
agencies. Assistant Superintendent Sheila Briggs outlined the theme for the 
event. Kate Gerson discussed the importance of standards-based education in 
increasing equity for disadvantaged ethnic groups. Eric Hirsch discussed the 
importance of materials, examined research findings about the impact of using 
materials, and considered recommendations for a strong adoption and 
implementation process. In a concluding session, three educators described 
local adoption and implementation processes used in Wisconsin. 


In response to the needs of stakeholders that do not generally use the 
cooperative educational service agencies for technical assistance, the 
Department of Public Instruction sponsored a five-part professional learning 
series in 2019 focusing on implementation of high-quality materials. This series 
included a two-day workshop facilitated by EdReports.org focusing on selection 
and adoption of materials, and four additional workshops facilitated by 
Instruction Partners focusing on their online guide. Instruction Partners (n.d.) 
consists of background information, two vignettes, a workbook, and a collection 
of resources. The online guide can be used by teams of teachers for selecting 
and implementing materials through three phases: selecting great materials; 
preparing to launch materials; and teaching with the materials and learning how 
to use them. 


Late in 2019, the Department of Public Instruction surveyed the state’s 421 
districts to identify materials they use in English language arts and 
mathematics, their selection processes, and types and frequencies of 
professional learning. Of 326 districts that responded to the survey, 
approximately 50 percent have a formal selection and adoption process. 


Late in 2018, Cooperative Educational Service Agency 4, serving 26 mostly 
rural districts in western Wisconsin, received a grant from the Department of 
Public Instruction for four districts to design a better decision-making process 
for selecting materials. The agency's staff facilitated meetings for teams from 
the four districts to consider EdReports.org reviews of mathematics and English 
language arts materials, as well as case studies on how districts and states 
conduct the materials adoption process. The teams used EdReports.org 
reviews to screen available materials and compile a list of three materials for 
mathematics and two materials for English language arts. Another 12 districts 
were invited to participate in the project in order to encourage publishers’ 
representatives to give presentations focusing on EdReports.org reviews of 
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their products and the needs of district adoption committees for information. 
Later, EdReports.org (2019d) composed a case study outlining the adoption 
process. 


Partnership with Learning Forward 


A professional association based at Oxford, Ohio, Learning Forward builds the 
capacity of leaders to establish and sustain effective professional learning. At its 
summer institutes held at Portland, Oregon, in July 2018, Learning Forward 
collaborated with EdReports.org, the Carnegie Corporation of New York, BSCS 
Science Learning, CenterPoint Education Solutions, the Institute for the Study 
of Knowledge Management in Education and UnboundEd to strengthen 
professional learning for implementing materials. 


Prior to the institutes, Learning Forward (2018) published a paper exploring the 
premise that good teachers can motivate and engage students by using high- 
quality materials. This contention was supported by three case studies of 
exemplary practice: implementation of EL Education’s K-5 Language Arts 
Curriculum at Hollis Innovation Academy in Atlanta; implementation of 
Connected Mathematics Program, published by Pearson, by Colonial School 
District in Delaware; and piloting of Core Knowledge Language Arts published 
by Core Knowledge Foundation, Wit & Wisdom published by Great Minds and 
the Read Aloud Project published by Student Achievement Partners by 13 
districts in Tennessee. Five lessons were learnt from evidence obtained in the 
case studies. First, selecting high-quality, aligned materials are of key 
importance. Second, using a standards-aligned material well requires skilful 
professional learning. Third, investing in leadership at the school and district 
levels is essential. Fourth, ensuring expert teacher leaders is important. Fifth, 
effective team learning is part of a larger instructional improvement and learning 
system. Connecting professional learning to high-quality materials, however, 
poses three challenges: aligning assessments, observations and curriculum; 
establishing sufficient regularly scheduled time and structure for professional 
learning communities and other teaching strategies; and applying change 
management strategies. Five actions were recommended to_ integrate 
professional learning and high-quality materials. First, build deeper knowledge 
about this issue. Second, assess the quality of the curriculum. Third, establish 
professional learning communities. Fourth, strengthen learning teams. Fifth, 
develop building- and team-level expertise. 


In May 2018, Learning Forward held a webinar, Building the Case for 
Connecting High-Quality Curricula and Team-Based Professional Development. 
Presenters, Stephanie Hirsh and Tracy Crow from Learning Forward, Eric 
Hirsch and Lauren Weisskirk, and Jody Guarino from Orange County 
Department of Education discussed research findings confirming the 
importance of aligned curriculum and materials, and using professional learning 
communities to ensure effective implementation. 


The four-day institutes were conducted by 21 facilitators from the six 
organisations as well as Ed Trust and Cherry Creek School District in Colorado. 
The program consisted of an introductory session attended by 227 participants 
on the first day, the choice of one of five sessions on the second and third days 
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conducted by a partner, and a concluding session for all participants on the last 
day. The introductory session focused on advancing equity and excellence 
through professional learning tied to curriculum implementation. EdReports.org 
focused on high-quality materials and aligned professional development. BSCS 
Science Learning focused on building capacity to implement the NGSS. 
UnboundEd focused on understanding the standards and shifts in secondary 
mathematics. The Institute for the Study of Knowledge Management in 
Education focused on adopting open educational practice to improve literacy- 
based curriculum at the secondary level. CenterPoint Education Solutions 
focused on strengthening assessment literacy. In the concluding session, 
Learning Forward focused on how team learning cycles and _ high-quality 
materials strengthen learning. 


Change process 


In a previous study, Watt (2019) found that EdReports.org developed, diffused 
and adopted a program successfully to provide evaluative information about 
high-quality, standards-aligned materials to the education community. The 
following discussion extends the findings reported previously by analysing the 
change process in EdReports.org program with reference to the four research 
questions. 


The results of the case study show that decision-making by the policy group 
was successful during the contingency and congruence modes of the stage of 
context evaluation in specifying planning decisions for the program. The 
professional background of the founder played a significant part in shaping 
decision-making within the policy group. Maria Klawe completed studies in 
mathematics at the University of Alberta, computer science at the University of 
Toronto, and worked on mathematics research at IBM’s Almaden Research 
Center at San Jose, California, and computer science at the University of British 
Columbia. After being appointed president of Harvey Mudd College at 
Claremont, California, Maria became involved in improving science and 
mathematics education in schools. A meeting with Geoffrey Cowan, former 
president of the Annenberg Foundation Trust at Sunnylands, prompted a 
suggestion to invite philanthropist, William Gates, to a retreat to discuss the 
opportunity that digital content offers to help teachers become more effective. A 
need for teachers to be able to use digital content in mathematics education 
effectively was raised by participants at the first retreat. Consensus was 
reached that lack of a trusted entity, which could judge whether digital content is 
effective, represented the most important factor impeding improvement in the 
quality of digital materials. Subsequent meetings of the Sunnylands 
Mathematics Strategy Group, funded by William Gates, focused on determining 
the design for a program to evaluate digital materials with the aim of 
establishing a national source of evidence for approaches that would lead to 
improvement in student performance. 


The results of the case study show that decision-making by a wide range of 
decision-makers during the stage of input evaluation occurred within a decision 
setting that involved large change requiring a broad range of political support, 
considerable time and a large amount of resources. Such large change required 
decision-makers to apply a planned change model involving many steps and 


21 


organisations over a relatively long span of time. Funds provided by various 
private foundations and support given by Education First in working with the 
newly appointed Board of Directors enabled a procedural design for the 
program to be specified in terms of staffing, time, budget requirements, potential 
procedural barriers and relevance of the design to program objectives. 
EdReports.org (2019a) outlined specific activities undertaken to meet program 
goals and objectives. A wide range of experts from district leaders and 
curriculum specialists to entrepreneurs and foundation directors were involved 
in workshops to structure the program for the new organisation based on two 
key principles. First, the expertise of educators was recognised as critical for 
analysing materials. Second, transparency in the review process was essential 
in establishing trust in the educational community. The process for reviewing 
materials was initiated by a nationwide survey, which showed that the program 
would be invaluable. A team of educators with expertise in standards and 
materials reviewed existing rubrics to design a rubric for use in the review 
process. While reviews of materials for mathematics and English language arts 
were underway, EdReports.org conducted a series of pilots with district and 
regional leaders to learn more about decision-making involved in the selection 
process. The development of a series of resources to assist educators to use 
data from reports, published on reviews of materials, was an outcome from the 
pilots. Partnerships that led to the California Curriculum Collaborative and the 
Nebraska Instructional Materials Collaborative induced EdReports.org to initiate 
an application programming interface that allows these organisations to freely 
extract report data and place these data within their specific contexts. 


The results of the case study show that decision-making by operations 
managers during the stage of process evaluation involved taking into account 
feedback about defects in the program received from stakeholders. Following 
criticism of the gateway procedure by several publishing companies, the 
National Council of Teachers of Mathematics and the National Council of 
Supervisors of Mathematics released an open letter in May 2015 outlining 
concerns that the gateway procedure excludes materials that fail to meet criteria 
for gateways one and two from a complete analysis. In June 2015, 
EdReports.org released four enhancements. First, all materials that partially 
meet expectations for focus and coherence at gateway one would be reviewed 
against the criteria for mathematical practice and rigour at gateway two. 
Second, evidence collection and scoring for the indicator determining whether 
students are assessed and held accountable for future grade level standards 
would be revised. Third, the EdReports.org website would offer more detailed 
rating visuals to illustrate the relative range of possible scores within reviews, 
and the evidence guides would be published on-line to allow teachers to 
conduct independent reviews. Fourth, publishers would be invited to share more 
background information about their materials, supplementary services they 
offer, and evidence of the effectiveness of their materials. In mid-2019, 
EdReports.org surveyed teachers, publishers, researchers and representatives 
of districts, states and non-profit organisations to identify stakeholders’ 
perceptions about the effectiveness of the review criteria, evidence guides, 
review process and the training program. Feedback collected from the survey 
was incorporated into the revised rubrics released in April 2020. 


The results of the case study show that decision-making by operations 
managers during the stage of product evaluation increasingly involved 
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measurement and interpretation of attainments in the program. In 2019 and 
2020, EdReports.org published its first reports. The first annual report used 
historical method to reconstruct the change process in the program, since its 
inception in 2015 up to 2018. A second report applied statistical analysis to 
measure the effects that reviews of English language arts and mathematics 
materials, conducted by EdReports.org, had on teachers’ use of these 
materials. The second annual report presented raw data to report on the 
effectiveness of the program, demographic characteristics of reviewers, and 
alignment of reviewed materials to standards. Assimilation of the program as an 
integral component of the American education system led state policymakers 
and education officials to acknowledge and accept that the program’s objectives 
have been accomplished. The case studies on the collaborative projects 
confirm that the goals and methodologies of its program have been adapted in 
California, Massachusetts and Nebraska, and applied to a more limited extent in 
Mississippi, Rhode Island and Wisconsin. The successful transfer and 
adaptation of the programs of these state education agencies to external 
settings suggests that personnel engaged in these contexts judged the 
programs to be effective in improving local decision making for selecting 
materials. 


Conclusion 


The findings of the evaluation show that decision-making by policy groups and 
operations managers was effective in planning, structuring and implementing 
the EdReports.org program. Furthermore, evidence from reports, recently 
published by EdReports.org, shows that the program is effective in meeting the 
needs of educators for guidance and advice about the selection and 
implementation of high-quality, standards-aligned materials. 


Collaboration and partnerships with other organisations and state education 
agencies show that the program has become institutionalised as part of regular 
educational programs in school systems in several states. Collaboration with 
Learning Forward emphasises the role of professional learning for implementing 
materials, while collaboration with UnboundEd and _ Instruction Partners 
supports the pivotal role of EdReports.org in standards-based education. 
Partnerships with Orange County Department of Education and Pivot Learning 
in California, as well as several state education agencies, have demonstrated 
its capability to transfer elements of its program to external settings. 
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